class: center, middle, inverse, title-slide # Automated Assessment
and Standards Alignment
of R Packages ### Mark Padgham & Noam Ross
rOpenSci & EcoHealth Alliance
Münster, Germany & New York, USA --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # rOpenSci: Transforming science through open data, software & reproducibility "*We help develop R packages for the sciences via community driven learning, review and maintenance of contributed software in the R ecosystem*" --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # rOpenSci & Software Peer Review "packages contributed by the community undergo a transparent, constructive, non adversarial and open review process." --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # rOpenSci & Software Peer Review: Why? - Feedback & improved software quality - Support from rOpenSci - Promotion of your software - Cross-linking with other software - Publications --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # rOpenSci & Software Peer Review: Scope Current scope includes software for: .left-column[ - data retrieval - data extraction - data munging - data deposition - data validation and testing - workflow automation - version control - citation management and bibliometrics ] .right-column[ - scientific software wrappers - field and laboratory reproducibility tools - database software bindings - geospatial data - text analysis ] --- class: center, top background-image: url(data:image/png;base64,#img/SloanLogo-1B-SMALL-Gold-Blue.png) background-size: contain background-position: 50% 50% -- # Expanding Software Peer Review -- .fontBIG[ Expand Scope<br>to include<br>Statistical Software ] --- class: center, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # Expanding Software Peer Review .fontBIG[ Expand Scope<br>to include<br>Statistical Software ] --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # Statistical Software Peer Review ## What is statistical software? --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # Statistical Software Peer Review ## What is statistical software? ### Methodology: - Empirical collation of abstracts from: - JSM conferences - *Journal of Statistical Software* - *Journal of Open Source Software* - Identify key statistical terms and represent as relational network - Analyse network structure & identify distinct clusters --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # Statistical Software Peer Review ## What is statistical software? ### Results: .left-column[ - Bayesian and Monte Carlo - Regression and Supervised Learning - Dimensionality Reduction, Clustering, and Unsupervised Learning - Exploratory Data Analysis (EDA) and Summary Statistics ] .right-column[ - Time Series Analyses - Machine Learning - Spatial Analyses - ~~Wrapper Packages~~ - ~~Network Analysis~~ - ~~Probability Distributions~~ - ~~Workflow Support~~ ] --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # Statistical Software Peer Review ## Our approach - General and category-specific standards - Standards compliance documented within code - Automated assessment tools --- class: left, top background-image: url(data:image/png;base64,#img/package-flow.png) background-size: contain background-position: 50% 50% --- class: center, middle, inverse # autotest ## github.com/ropensci-review-tools ## search: "ropensci autotest" --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package: Why? ## rOpenSci Software Peer Review - A peer-review system integrating practices from software development and academic publishing - 7 years experience, hundreds of packages, authors and reviewers - Designed to improve software and expand a community of practices, via transparent, constructive, non-adversarial and open review --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package: Why? ## rOpenSci Software Peer Review - Reviewers often the first "external" users of submitted packages - Quickly uncover bugs, distracting reviews from higher-level focus on software quality - A tool to help uncover bugs prior to submission should increase software quality --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package: How? - A form of "mutation testing" - Extracts documented example code to identify all inputs to, and outputs from, all functions - Mutates those inputs - Also matches input and output types to descriptions --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package 1/3: Set up a package to test: ```r path <- file.path (tempdir (), "demo") usethis::create_package (path) fs::dir_tree (path) ``` ``` ## /tmp/RtmpRCSx84/demo ## ├── DESCRIPTION ## ├── NAMESPACE ## └── R ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package 2/3: Add a function: ```r #' my_function #' #' @param x An input #' @return Something else #' @export my_function <- function (x) { return (x + 1) } ``` - save to `./R/myfn.R` - run `roxygen::roxygenise()` to create documentation (`/man`) file --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package 3/3: `autotest` it: ```r library (autotest) a <- autotest_package (path, test = TRUE) ``` ``` ## ## ── autotesting demo ── ## ``` ```r class (a) ``` ``` ## [1] "autotest_package" "tbl_df" "tbl" "data.frame" ``` ```r dim (a) ``` ``` ## [1] 1 9 ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package 3/3: `autotest` it: ```r library (autotest) a <- autotest_package (path, test = TRUE) print (a) ``` ``` ## type test_name fn_name parameter parameter_type ## 1 warning fn_without_example my_function <NA> <NA> ## operation ## 1 Identify functions without documented examples ## content test yaml_hash ## 1 This function has no documented example TRUE <NA> ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package 3/3: `autotest` it: ```r library (autotest) a <- autotest_package (path, test = TRUE) a$content ``` ``` ## [1] "This function has no documented example" ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ```r #' my_function #' #' @param x An input #' @return Something else #' @export my_function <- function (x) { return (x + 1) } ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ```r #' my_function #' #' @param x An input #' @return Something else #' @export #' @examples #' y <- my_function (x = 1) my_function <- function (x) { return (x + 1) } ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ```r library (autotest) a <- autotest_package (path, test = TRUE) ``` ``` ## ## ── autotesting demo ## ✔ [1 / 1]: myfunction ``` ```r a$content ``` ``` ## [1] Parameter [x] is not specified as integer, yet ## only used as such; please use '1L' for integer, ## or 1.0 for non-integer values. ## [2] Parameter [x] of function [my_function] is only used ## as a single numeric value, but responds to vectors of ## length > 1 ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ```r #' my_function #' #' @param x An input #' @return Something else #' @export #' @examples #' y <- my_function (x = 1) my_function <- function (x) { return (x + 1) } ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ```r #' my_function #' #' @param x An input #' @return Something else #' @export #' @examples #' y <- my_function (x = 1L) my_function <- function (x) { checkmate::assertInt (x) return (x + 1) } ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ```r library (autotest) a <- autotest_package (path, test = TRUE) ``` ``` ## ## ── autotesting demo ## ✔ [1 / 1]: myfunction ``` ```r a$content ``` ``` ## [1] Parameter [x] permits unrestricted integer inputs, ## yet does not document this; please add 'unrestricted' ## to parameter description. ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ```r #' my_function #' #' @param x An input #' @return Something else #' @export #' @examples #' y <- my_function (x = 1L) my_function <- function (x) { checkmate::assertInt (x) return (x + 1) } ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ```r #' my_function #' #' @param x An unrestricted integer input #' @return Something else #' @export #' @examples #' y <- my_function (x = 1L) my_function <- function (x) { checkmate::assertInt (x) return (x + 1) } ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ```r library (autotest) a <- autotest_package (path, test = TRUE) ``` ``` ## ## ── autotesting demo ## ✔ [1 / 1]: myfunction ``` ```r a ## NULL ``` -- ### When `autotest` works, it should do nothing! --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ## Which tests are run? `test = FALSE` ```r a <- autotest_package (path, test = FALSE) a ``` ``` ## # A tibble: 9 x 9 ## type test_name fn_name parameter parameter_type operation content test ## <chr> <chr> <chr> <chr> <chr> <chr> <chr> <lgl> ## 1 dummy int_range my_func… x single integer Ascertain … Should e… TRUE ## 2 dummy int_as_nu… my_func… x single integer Integer va… (Should … TRUE ## 3 dummy single_pa… my_func… x single integer Length 2 v… Should t… TRUE ## 4 dummy return_su… my_func… (return … (return objec… Check that… <NA> TRUE ## 5 dummy return_va… my_func… (return … (return objec… Check that… <NA> TRUE ## 6 dummy return_de… my_func… (return … (return objec… Check whet… <NA> TRUE ## 7 dummy return_cl… my_func… (return … (return objec… Compare cl… <NA> TRUE ## 8 dummy par_is_do… my_func… x <NA> Check that… <NA> TRUE ## 9 dummy par_match… my_func… x <NA> Check that… <NA> TRUE ## # … with 1 more variable: yaml_hash <chr> ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ## Which tests are run? `test = FALSE` ```r a <- autotest_package (path, test = FALSE) a$test_name ``` ``` ## [1] "int_range" "int_as_numeric" ## [3] "single_par_as_length_2" "return_successful" ## [5] "return_val_described" "return_desc_includes_class" ## [7] "return_class_matches_desc" "par_is_documented" ## [9] "par_matches_docs" ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ## Which tests are run? `test = FALSE` ```r #' my_function #' #' @param x An unrestricted integer input #' @return Something else #' @export #' @examples #' y <- my_function (x = 1L) my_function <- function (x) { checkmate::assertInt (x) return (x + 1) } ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ## Which tests are run? `test = FALSE` ```r #' my_function #' #' @param x An unrestricted integer input #' @return Something else #' @export #' @examples #' y <- my_function (x = 1.) my_function <- function (x) { checkmate::assertNumber (x) return (x + 1) } ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ## Which tests are run? `test = FALSE` ```r a <- autotest_package (path, test = FALSE) a$test_name ``` ``` ## [1] "double_is_int" "trivial_noise" ## [3] "single_par_as_length_2" "return_successful" ## [5] "return_val_described" "return_desc_includes_class" ## [7] "return_class_matches_desc" "par_is_documented" ## [9] "par_matches_docs" ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package A real example: `autotest`ing the `stats` package ```r a <- autotest_package (package = "stats", functions = "var", test = FALSE) # default ``` - `package` argument can be name of an installed package, or path to local source directory - `functions` argument can restrict tests to specified functions only --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package A real example: `autotest`ing the `stats` package ```r a <- autotest_package (package = "stats", functions = "var", test = FALSE) # default ``` ``` ## ## ── autotesting stats ## ✔ [1 / 4]: var ## ✔ [2 / 4]: cor ## ✔ [3 / 4]: cov ## ✔ [4 / 4]: cov ``` ```r dim (a) ``` ``` ## [1] 150 9 ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package A real example: `autotest`ing the `stats` package ```r a <- autotest_package (package = "stats", functions = "var", test = TRUE) # takes 20-30 seconds ``` ``` ## ## ── autotesting stats ## ✔ [1 / 4]: var ## ✔ [2 / 4]: cor ## ✔ [3 / 4]: cov ## ✔ [4 / 4]: cov ``` ```r dim (a) ``` ``` ## [1] 15 9 ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package A real example: `autotest`ing the `stats` package ``` ## # A tibble: 15 x 9 ## type test_name fn_name parameter parameter_type operation content test ## <chr> <chr> <chr> <chr> <chr> <chr> <chr> <lgl> ## 1 warni… par_is_de… var use <NA> Check tha… "Example… TRUE ## 2 warni… par_is_de… cov y <NA> Check tha… "Example… TRUE ## 3 diagn… single_ch… cor use single charac… upper-cas… "is case… TRUE ## 4 diagn… single_ch… cor method single charac… upper-cas… "is case… TRUE ## 5 diagn… single_ch… cor use single charac… upper-cas… "is case… TRUE ## 6 diagn… single_ch… cor method single charac… upper-cas… "is case… TRUE ## 7 diagn… single_ch… cov use single charac… upper-cas… "is case… TRUE ## 8 diagn… single_ch… cov method single charac… upper-cas… "is case… TRUE ## 9 diagn… single_ch… cov use single charac… upper-cas… "is case… TRUE ## 10 diagn… single_ch… cov method single charac… upper-cas… "is case… TRUE ## 11 diagn… single_ch… cov use single charac… upper-cas… "is case… TRUE ## 12 diagn… single_ch… cov method single charac… upper-cas… "is case… TRUE ## 13 diagn… vector_to… var x vector Convert v… "Functio… TRUE ## 14 diagn… vector_to… var x vector Convert v… "Functio… TRUE ## 15 diagn… vector_to… var y vector Convert v… "Functio… TRUE ## # … with 1 more variable: yaml_hash <chr> ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package A real example: `autotest`ing the `stats` package - Use `DT` package to inspect results! ```r a <- autotest_package (package = "stats", functions = "var", test = TRUE) # takes 20-30 seconds DT::datatable (a) ``` --- class: left, top background-image: url(data:image/png;base64,#img/stats-var-TRUE.png) background-size: contain background-position: 50% 50% --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package ## Which tests are run? `test = FALSE` ```r a <- autotest_package (package = "stats", functions = "var", test = FALSE) dim (a) ``` ``` ## [1] 150 9 ``` ```r DT::datatable (a) ``` --- class: left, top background-image: url(data:image/png;base64,#img/stats-var-FALSE1.png) background-size: contain background-position: 50% 50% --- class: left, top background-image: url(data:image/png;base64,#img/stats-var-FALSE2.png) background-size: contain background-position: 50% 50% --- class: left, top background-image: url(data:image/png;base64,#img/stats-var-FALSE3.png) background-size: contain background-position: 50% 50% --- class: left, top background-image: url(data:image/png;base64,#img/stats-var-FALSE4.png) background-size: contain background-position: 50% 50% --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'autotest' package A real example: `autotest`ing the `stats` package ```r a <- autotest_package (package = "stats", functions = "var", test = TRUE) ``` - `stats` package is algorithmially robust - But `autotest` reveals a few gaps in documentation! --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # Statistical Software Peer Review ## Our approach - General and category-specific standards - Standards compliance documented within code - Automated assessment tools --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # Statistical Software Peer Review ## Our approach - General and category-specific standards - Standards compliance documented within code - ~~Automated assessment tools~~ --- class: center, middle, inverse # Statistical Standards ## Online Book Contains all General and Category-Specific Standards `stats-devguide.ropensci.org` --- class: left, top background-image: url(data:image/png;base64,#img/devguide-stds-G1.png) background-size: contain background-position: 50% 50% --- class: left, top background-image: url(data:image/png;base64,#img/devguide-stds-G2.png) background-size: contain background-position: 50% 50% --- class: left, top background-image: url(data:image/png;base64,#img/devguide-stds-G3.png) background-size: contain background-position: 50% 50% --- class: left, top background-image: url(data:image/png;base64,#img/devguide-stds-G4.png) background-size: contain background-position: 50% 50% --- class: left, top background-image: url(data:image/png;base64,#img/devguide-stds-BS1.png) background-size: contain background-position: 50% 50% --- class: left, top background-image: url(data:image/png;base64,#img/devguide-stds-BS2.png) background-size: contain background-position: 50% 50% --- class: left, top background-image: url(data:image/png;base64,#img/devguide-stds-BS3.png) background-size: contain background-position: 50% 50% --- class: center, middle, inverse # Statistical Standards ## 40 General Standards &<br>265 Category-Specific Standards<br>in 7 Categories ## Four New Categories in Progress --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # Statistical Software Peer Review ## Our approach - General and category-specific standards - Standards compliance documented within code - ~~Automated assessment tools~~ --- class: left, top background-image: url(data:image/png;base64,#img/package-flow.png) background-size: contain background-position: 50% 50% --- class: center, middle, inverse # The 'srr' package ## `srr` = "Software Review Roclets" ## github.com/ropensci-review-tools ## search: "ropensci srr" (A "roclet" is a documentation engine,<br>or "doclet", for the **R** language.) --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'srr' package: Why? - Software must comply with detailed, category-specific standards - `srr` allows standards compliance to be documented within the software itself --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'srr' package ## documentation in R with 'roxygen2' ```r #' Function description #' #' @rdtitle NewFunction #' @param x First input #' @param y Second input #' @return Something different #' @export myfunction <- function (x, y) { # ... } ``` -- Run `roxygen2::roxygenise()` to turn that into `/man` file `NewFucntion.Rd` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'srr' package ## documentation in R with 'roxygen2' - `roxygen2` is extensible, and allows custom "roclets" - Everything following `@` is a "tag" - Each tag has a "roclet" - Each roclet has a generic `process` method, and an `output` method to output information to the `.Rd` file - Tags with `output = NULL` do not appear in package documentation --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'srr' package - `srr` introduces 3 new tags for documenting standards: - `@srrstats` for adherence to standards - `@srrstatsNA` for non-applicable standards - `@srrstatsTODO` for standards yet to be addressed --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'srr' package - The `srr_stats_roxygen()` function dumps full text of all standards in file `R/srr-stats-standards.R` - All are initially tagged `@srrstatsTODO` - Standards are addressed by changing tag to `@srrstats` and moving to location within code where standard is addressed - `srr_report()` function generates a `.html` report of standards compliance --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'srr' package ```r #' Function description #' #' @rdtitle NewFunction #' @param x First input #' @param y Second input #' @return Something different #' @export myfunction <- function (x, y) { # ... } ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'srr' package ```r #' Function description #' #' @rdtitle NewFunction #' @param x First input #' @param y Second input #' @return Something different #' @export #' #' @srrstats {G1.1} This standard is addressed here through ... myfunction <- function (x, y) { # ... } ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # The 'srr' package ```r library (srr) path <- srr_stats_pkg_skeleton() roxygen2::roxygenise (path) ``` ``` ## ℹ Loading demo ## Re-compiling demo ## ─────────────────────────────────────── rOpenSci Statistical Software Standards ────────────────────────────────────── ## ## ── @srrstats standards: ## * [G1.1, G1.2, G1.3, G2.0, G2.1] in function 'test_fn()' on line#11 of file [R/test.R] ## * [RE2.2] on line#2 of file [tests/testthat/test-a.R] ## * [G2.3] in function 'test()' on line#6 of file [src/cpptest.cpp] ## * [G1.4] on line#17 of file [./README.Rmd] ## ## ── @srrstatsNA standards: ## * [RE3.3] on line#5 of file [R/srr-stats-standards.R] ## ## ── @srrstatsTODO standards: ## * [RE4.4] on line#14 of file [R/srr-stats-standards.R] ## * [RE1.1] on line#11 of file [R/test.R] ## * [G1.5] on line#17 of file [./README.Rmd] ## ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── ``` --- class: left, top background-image: url(data:image/png;base64,#img/icon_lettering_color_large-faded.png) background-size: contain background-position: 50% 50% # Statistical Software Peer Review - Now accepting submissions - Use of `autotest` will enable reviews to better focus on substantive issues of software design and quality - `srr` provides a comprehensive method for documenting standards compliance within code itself - Associated `html` reports facilitate rapid review of standards compliance - Result should be high quality statistical software with open and demonstrable compliance with standards